The invention relates generally to data management systems and, more particularly, to a system and method for efficient data compression and data storage in a data management system.
In many industries, data management systems are important for efficient monitoring, sampling, storage, handing, processing, and analysis of data relating to certain equipment, processes, and so forth. For example, certain data management systems are useful for monitoring various processes at different times and locations by collecting operational data or parameters, which are useful for subsequent analysis of the monitored processes. In certain processes, the operational data or parameters include pressure, temperature, sound, velocity, fluid flow rate, or any other measurable physical, chemical, or biological parameters. The subsequent analysis may include a performance evaluation, an error analysis, a cost analysis, a failure prediction or life expectancy analysis, or another desired evaluation of the various processes.
In many of these analyses, the data management system is limited based on data sampling rates, data storage capacity, data compression efficiency, and other bottlenecks. The accuracy of these analyses often depends on the degree of data sampling, i.e., a higher level of data sampling results in greater accuracy of the analysis. Unfortunately, limitations of data storage capacity and compression efficiency often result in lower than desired rates of data sampling, thereby reducing the accuracy of the particular analysis. In process control applications, for example, data is collected from one or more parallel processes at very frequent intervals of time (e.g. one per second or one per minute), which leads to an immensely large amount of collected data. As a result, data compression is important for reducing the storage consumption of this collected data, while also maintaining the important portions of the collected data for use in the desired analysis.
Existing data compression techniques reduce the storage consumption of the collected data, yet these techniques sacrifice data accuracy for ease of storage. For example, one existing data compression technique involves combining and averaging a group of sequential data to create one average data point. However, by averaging, the data management system loses variations above and below the average data point, which may lead to inaccurate predictions based on the averaged data. Similarly, other existing data compression techniques may discard valid and significant data to save or open storage space for subsequent incoming data.
Therefore, there exists a need for an efficient data compression technique that provides for accurate data analysis and predictions.